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Summary 


The demands of large-scale distributed (LSD) systems is increasing. There are two main influential factors 
that encourage employing such applications in many domains: the advances in the Internet and the Intranets 
technology and the economical and performance benefits of distributed applications, in general. Examples of 
LSD systems include large-scale collaborative distance learning, video conferencing, group-ware editinq 
distributed transaction systems and distributed interactive simulation. LSD systems involve a larqe number 
of users or application entities which are geographically dispersed over interconnected LANs (i.e. Intranets) 
° r o y er ^ AN (f; 1 - ,nternet )- Such applications enable more interaction and resource sharing that go beyond 
the imitation of long geographical distances. For instance, in large-scale distance learning and traininq 
applications, a large number of students who may be far distant from each other can share the same benefit 
of attending a course, and exchanged experiences regardless of the distance or the number of participants. 
Similarly, database systems (such as banking systems) can increase its computational and data storaqe 
capacity by utilizing large-scale transaction systems. y 

D r 6 uM-l he ^ trib , Uted nature and the lar 9© number of participants or application entities in LSD system 
reliability and performance of such applications become critical issues. The wide geographical distribution 
and large interaction of such applications may sometimes increase the possibility of failures/errors or 
performance bottlenecks. Unlike centralized or isolated applications, LSD systems are likely to deal with 
different environments that may be dynamically changing. For these reasons, observing the run-time 
behavior of LSD systems is essential to discover reliability and performance problems and initiate the proper 
reactions to alleviate these problems. These actions are performed either at run-time such as fault recovery 
and applications steenng or at development-time such as fixing bugs and design enhancements. 

The LSD system developers or managers (could be human or software components) require feedback 
information on the system behavior for testing and debugging purposes or for fault recovery and 
performance tuning procedures. Monitoring LSD systems is an effective means to observe applications at 
run-time and provide this feedback information to the system developers and system managers in order to to 
improve reliability, robustness and performance of LSD systems. 

In LSD system a large number of events is generated by the system components (e.g. processes) durinq its 
execution or interaction with an external objects such as users or other applications. These events represent 
the ron-time behavior of LSD systems. In centralized or isolated systems, developers express these events 
via print” statements or via using any generic debugger tool (e.g. gdb) for monitoring and inspecting the 
app ica ion e avior. However, in LSD systems, monitoring is much more complex because events can be 
concurrent and distributed in the application environment. In other words, unlike centralized or isolated 
systems, events that may represent application errors or bugs may be correlated and distributed over many 
ocations. For example, application errors related to the communication operations obviously involve 
obseiving the sendees) and the receivers(s). Similarly, the knowledge of application performance is also 
distributed in the application environment. For instance, calculating the average load of the system must 

involve all participant machines. Thus, concurrency and distribution of events makes the tasks of testinq and 
management decisions much harder. y 

Unlike centralized or isolated systems, in LSD environment, errors/failures events (due to software buqs or 
improper implementation) or events that convey the application status can be dispersed over many different 
locations and application entities which makes the task of testing and debugging or management decisions 
process (e.g. fault recovery or performance tuning procedures) much harder to achieve in LSD systems. 

Furthermore, the monitored events, either simple (local) or complex (global) events, which may encounter 

th« °t ° Ck Skewing P roblems - The large volume of event traffic which flows in the system 
may swamp the monitoring process. y 

high-performance architecture for monitoring interesting local and global 
events of LSD systems and disseminating the monitoring information dynamically to the subscribing 
“'? 9 a P pllcatl ° ns s “ ch as distributed debugging and reactive control tools which subsequently would 
effectively improve the robustness (via debugging), the reliability (via fault recovery), and the performance 
(via performance tuning and application steering). Although our main focus in this proposal is on thSe 
monitoring applications: reliability and performance tuning, the proposed monitoring architecture can be 



used for many other monitoring applications such as security and correctness checking. We show how the 
design can satisfy the work objectives of supporting a scalable, high-performance, dynamic, flexible and 
non-intrusive architecture for monitoring large-scale distributed systems. Various optimization techniques 
are proposed for the event filtering mechanism to increase the scalability and the performance of the 
monitoring process and minimize overhead on the computation and the network resources which in turn 
reduces the intrusiveness of the monitoring operations. We show an emphasis in studying and designing the 
filtering component of the monitoring architecture, since the event filtering mechanism is an intrinsic 
component that has a significant impact on the monitoring performance and scalability. Our monitoring 
architecture provides a simple mechanism to prepare (i.e. instrument) the monitored applications for 
capturing and reporting generated events with a minimal intervention from the application developers. It also 
supports a dynamic monitoring by enabling the monitoring applications (e.g. managers) to dynamically 
change their monitoring requests (called subscription) at run-time. The monitoring architecture provide a 
flexible service by providing priority-based event monitoring service where events are processed based on 
their priorities and adjustable instrumentation mechanism (e.g. adjusting the event reporting rate). 

Although there is a considerable amount of research work on monitoring in general and monitoring 
distributed and parallel applications in specific, the proposed systems are insufficient for satisfying our 
design objectives and supporting a scalable, high-performance, dynamic, flexible and non-intrusive 
monitoring architecture for large-scale distributed systems. 


Debugging Example in Monitoring IRI Application 

In this section, we show an example of using the monitoring architecture fortesting and debugging IRI 
application. The goal of this example is to illustrate more the monitoring process and show how our 
monitoring architecture can be used effectively and easily for testing and debugging purposes. Many other 
examples of this application or other monitoring applications can be developed by following the a similar 
procedure. 

The IRI application is collaborative distributed applications where exchanging messages is a major activity 
in the system. The Reliable Multicast Protocol Server (RMPS) is the main communication module in IRI 
application. In such message-based applications, it is very likely to encounter a mismatch in message sizes 
because of errors in sending/receiving operations such as type mismatch between the sender and 
the receivers) or the kernel alignment of sent packetjgif] . Therefore, debugging “send" and “receive” 
operations is highly desired in such message-based distributed application such as IRI as well as any 
client/server application, in general. Thus, our debugging example is monitoring the send and receive 
activities (events) in RMPS and reporting information about any mismatch in sent and received message 
sizes. In this monitoring example, the consumer(s) could be the developers) and RMPS entities are the 
event producers. This example represent a composite event since the knowledge of sent/received message 
sizes is distributed in the IRI virtual classroom. In the following, we describe how this monitoring example is 
constructed and processed in IRI application environment. 


Filter Subscription in HFSL: A developer may define his/her filter subscription as follows: 

FILTER= [tex2html_wrap_inline2827] 

[(MSend.lPsrc = [tex2html_wrapjnline2831] = [tex2html_wrapjnline2833] 

[tex2html_wrap_inline2835] 

The MSend and MRec are the multicast send and receive events respectively. The Report_Mismatch action 
is a function or program that will send the monitoring information to the developer reporting the occurrence 
of this event. 

Event Specifications in HESL: The send multicast event (called MSend) and the received multicast event 
(called MRec) may be specified respectively as follows: 

EVENT= [tex2html_wrap_inline2837] Module_Name=RMPS, Func_Name=Send, NULL, Immediate; 
IPdest=224.\V, IPsrc=ANY [tex2html_wrapjnline2839] MSend. 



EVENT= [tex2html_wrap_inline2837] Module_Name=RMPS, Func_Name=Receive, NULL, Immediate; 
IPdest=224. IPsrc=ANY [tex2html_wrap_inline2839] MRec. 

RMPS Instrumentation: The RMPS send and receive routines are instrumented to report information about 
any send and receive events according to the event specifications: event name, message sequence 
number, IP source address and size of sent and received message. These fields are used in the filter 
definition (program). 

This shows the filtering internal representation after the filter is decomposed and distributed between the 
monitoring agents. The filter is decomposed to three subfilters: FI which detects receiving multicast events 
and forwards it to its DMA, F2 which detects sending multicast events and forwards them to all DMAs, F3 
which is responsible for evaluation the filter expression by comparing the receiving and the sending 
multicast events. If the composite event represented by Msg_Mismatch_FILTER is detected, then F3 will 
forward the monitoring information to the developer(s) as requested. Notice that sending and receiving 
events can take a place at any machine (RMPS resides on every machine in IRI application). Therefore, 
based on the Environment Specifications, the FI and F2 subfilters are delegated to all LMAs in the virtual 
classroom. However, F3 are delegated to the DMAs which get the MSend and MRec events and evaluate 
the filter expression accordingly. The extracting layer in th figure is just to forward only the relevant 
information and reduce the event traffic. Finally, the requested monitoring information is forwarded to one or 
more developers based on their subscription. This simple example shows how a developer can effectively 
define and monitor IRI application activities at run-time and collect the desired debugging information from 
various locations in the application environment without the hassle of analyzing multiple traces or inspecting 
the application entities at different locations. 
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